The impact of users’ cognitive function on evaluator perceptions of usability

To explore the association between the user’s cognitive function and usability reported by the evaluator. A cross-sectional study was conducted with a community-based sample. Data about participants’ age, sex, education, sleep quantity, subjective memory complaints, and cognitive function were collected. A usability session was conducted to evaluate a digital solution called Brain on Track. Independent linear-regression analyses were used to explore univariable and multivariable associations between evaluator-reported usability assessment and the users’ cognitive function, age, sex, education, sleep quantity, and subjective memory complaints. A total of 238 participants entered this study, of which 161 (67.6%) were females and the mean age was 42 (SD 12.9) years old. All variables (age, education, sleep quantity, subjective memory complaints and cognitive function) except sex were significantly associated with evaluator-reported usability in the univariable analysis (p < 0.05). Cognitive function, age, education, and subjective memory complaints remained significant in the multivariable model (F = 38.87, p < 0.001) with an adjusted R2 of 0.391. Cognition scores alone showed an adjusted R2 of 0.288. This work suggests that cognitive function impacts evaluator reported usability, alongside other users’ characteristics and needs to be considered in the usability evaluation.

www.nature.com/scientificreports/ poorer in these individuals (they complete fewer tasks, take longer to complete them or need more help) when compared with older adults without cognitive impairment 19,20 . These findings suggest that cognitive functioning might influence evaluators perceived usability, which is based on the performance of the individual while interacting with the digital solution during the usability evaluation. Therefore, this study aims to explore the association between cognitive function and usability reported by the evaluator in a community of cognitively unimpaired dwelling adults.

Methods
This is a cross sectional study approved by the Ethics Committee of the Centro Hospitalar de Entre o Douro e Vouga, EPE, Portugal (process number CA-0114/16-0c, date 25 Jan 2016). All participants signed an informed consent prior to entering the study and all methods were performed in accordance with the relevant guidelines and regulations.

Participants and inclusion and exclusion criteria.
A community-based sample of residents of the region of Águeda, Aveiro, Portugal was involved in the study. To recruit participants, the study was publicized in the local media, in local dissemination actions and through leaflets placed at public establishments. Also, institutions linked to the public administration, industry, education, and the social area of the municipality were contacted, and employees and customers were invited to participate in the study. To be included, participants had to be 18 years old or older, be able to use a computer independently and be able to read in Portuguese. Participants were excluded if having cognitive deficit as assessed by the Montreal Cognitive Assessment (MoCA) as described below in the Procedures section.
Procedures. Participants who agreed to enter the study were asked to provide data regarding age, sex, years of formal education, sleep quantity, subjective memory complaints, and cognitive function. Sleep quantity was evaluated by self-reporting of the average hours of sleep per night on the last month. Subjective memory complaints were assessed using the Subjective Memory Complaints Scale (SMC). The SMC consists of ten items 21 . Each individual item is scored zero (in case of absence of complaints), one, two or three points (in case of maximum complaints) depending on the severity of the complaint. Total score varies between zero and of 21 points. The European Portuguese version of this scale is valid and reliable 22 . Values greater than or equal to four points indicate significant memory complaints and values less than or equal to three indicate no relevant memory complaints 22 .
Cognitive function was assessed with the MoCA, which evaluates different cognitive domains, namely executive function; visuospatial capacity; memory; attention, concentration and working memory; language and temporal and spatial orientation. It has been shown to have high sensitivity (90%) and specificity (87%) for detecting mild cognitive impairment 23 . It is validated for the Portuguese population 24 and total score ranges from 0 to 30 points, with higher scores representing better cognitive performances 23 . The cut off points for cognitive deficit vary between 16 and 27, depending on the participants age and education level, as defined by the Portuguese normative data 25 . According to the normative data for the Portuguese population, the MoCA scores are expressed as the means ± standard deviation (SD), and those of the distributions that are below 1 SD, 1.5 SDs, and 2 SDs can be considered as cutoff points for possible cognitive impairment 25 . To determine the MoCA cutoff points, the age and education are considered, and the SD used for this study was 1.5. MoCA was selected because it is a highly sensitive tool for early detection of mild cognitive impairment (MCI) and is widely used in research and clinical practice. Besides that, MoCA is validated for Portugal and has normative data for the Portuguese population. MoCA was used both to screen for inclusion and to characterize the cognitive function of participants who entered the study.
Usability evaluation. Usability evaluation took place in a single session. Participants were asked to use a digital solution called Brain on Track. This digital solution is a computerized web-based self-administered test intended for cognitive testing that allows monitoring of the cognitive performance of users 20,21 . This solution has gone through a demanding process of testing and validation and is on the market since 2013.
The usability session consisted of three parts: 1. Pre-test-The evaluator explained all the procedures of the study.
2. Test-The evaluator explained how to use the digital solution Brain on Track and the participant interacted freely with it for 25 min. The evaluator helped the participant whenever the participant had doubts about any feature of the system and observed the interaction. 3. Post-test-The evaluator filled the ICF based Usability Scale I (ICF-US I) 26 . The ICF-US I is a usability rating scale that can be used as a tool for the usability evaluation based on the evaluator's opinion 26 . This scale was used in this study because usability based on the evaluator's perspective showed higher correlation with indicators of user performance than usability based on the user's perspective 2 . The ICF-US I is composed of 10 items associated with different usability principles that are classified as facilitators or barriers. It is based on the International Classification of Functioning, Disability and Health (ICF) 27  Statistical analysis. All data analyses were performed using SPSS 28.0 for Windows (SPSS Inc, Chicago, IL). Mean and standard deviation and count and proportion were used to describe continuous and categorical variables, respectively. To determine possible factors associated with evaluator reported usability, independent linear-regression analyses were used to explore univariable and multivariable associations between the independent variables (age, sex, education, sleep quantity, subjective memory complaints and MoCA scores) and usability based on the opinion of the evaluator as the dependent variable. Normality was checked by visual inspection of the plots and no major violations were noted for the independent variables. The univariable analyses was used to identify the variables that would enter the multivariable model, and p ≤ 0.10 was required, except for sex and age, which were forced into the multivariable model. The multivariable analyses were performed using the enter, stepwise, backward and forward models and as the Adjusted R-squared of the models varied less than 0.01, the results of the stepwise model were reported. The variables that entered into the models were checked for multicollinearity using variance inflation factor (VIF) ≤ 5 and the respective tolerance value (> 0.2). Level of significance was set at p < 0.05.

Results
A total of 238 participants entered this study, of which 161 (67.6%) were females. The mean age was 42 (SD 12.9) years old (range 20-76 years old). Sample characteristics are presented in Table 1.
The mean (± SD) score of the ICF-US I was 25.38 (± 6.89), which indicates that the digital solution Brain on Track was considered a big facilitator. Of the 238 participants, 230 (96.6%) scored above the cut off (10 points) for considering the application a facilitator. The remaining 8 participants scored the digital solution below the cut off indicating that the application was considered a barrier.
Univariable associations with evaluator reported usability. All variables (education, sleep quantity, subjective memory complaints and cognitive function) were significantly associated with usability (Table 2) in the univariable analysis.
Multivariable associations with evaluator reported usability. Education, sleep quantity, subjective memory complaints, MoCA, sex and age were all include in the multivariable analysis as independent variables. Of these, age, education, subjective memory complaints and cognitive function remained significant in the multivariable model (F = 38.87, p < 0.001) with an adjusted R 2 of 0.391 (Table 3). MoCA scores alone showed an adjusted R 2 of 0.288.

Discussion
This study explored the association between cognitive function and usability reported by the evaluator in a community-based sample and results suggest that cognitive function impacts evaluator reported usability, alongside with age, education and subjective memory complaints. Age, mean (± SD) 42 (± 12.9) Years of formal education, mean (± SD) 14 (± 3.9) Hours of sleep, mean (± SD) 6.8 (± 1.1) SMC total (0-21), mean (± SD) 3.9 (± 2.8) MoCA total (0-30), mean (± SD) 27.17 (± 2.1) www.nature.com/scientificreports/ Cognitive function, accounted for most of the variance of usability explained by the multivariable model, which in total explained around 39% of the usability variance. The remaining 61% may be explained by the intrinsic usability of the system, and other variables that are yet to be studied.
Studies have suggested that using digital solutions, such as personal computers 29 or tablets 30 help maintain or enhance cognitive functions, such as memory, processing speed and attention. We are not aware of previous studies exploring the association between cognitive function and the perceived usability of the study neither in the evaluator's perspective nor in the user's perspective. However, for a user to be efficient in the use of a certain digital solution, several cognitive functions are involved, including concentration, attention, and memory 31 . Conceivably, the better the user's ability in these areas, the easier the interaction with this type of technologies will be. On the other hand, those users with lower cognitive ability may have more difficulty interacting with complex digital solutions. The findings of the present study suggest that assessing users' cognitive function in usability studies is relevant and raise important questions, namely to what extent is it necessary to differentiate the interfaces according to the cognitive skills of the users? Another important issue is the role of training, as the association between cognitive function and usability might decrease with continued use, but individuals with lower cognitive skills might require longer periods of training to be able to effectively and efficiently use a digital solution. Also, this study results reinforce the importance of applying usability principles in the design of digital solutions such as minimize memory load, meaning that users should be able to recognize rather than recall or use minimalist design to avoid distractions and favor concentration 32,33 .
It is important to note that this study did not include individuals with a previous diagnosis of mental or neurological diseases, but people living in the community with no identified cognitive impairment.
A limitation of the present study is the fact that digital literacy was not considered, and it is one of the variables mentioned in the literature as influencing usability 10,34 . This variable is difficult to assess comprehensively, as it is not just about being proficient in the use of technology, but also about its ethical and responsible use and it is related to social and cultural aspects 35 . Another study limitation is the fact that neither user self-perceived usability nor objective measures of usability were evaluated, and it would be interesting to explore whether they are also associated with cognitive function. The ICF-US I was used to study the association with usability reported by the evaluator as it is a reliable tool that overcomes the difficulty reported in the literature related to the fact that the opinion of the users, collected through the filling of self-perceived generic usability scales, do not fully reflect users' performance 26,36 .

Conclusion
Cognitive function seems to be associated to the usability of digital solutions and influence the human-technology interaction. The results of this study suggest that cognition has an even more pronounced influence in usability than other user characteristics such as age or education.

Data availability
All data needed to evaluate the conclusions in the paper are present in the paper. Additional data related to this paper may be requested from the authors.